Visual Reasoning with Natural Language

Natural language provides a widely accessible and expressive interface for robotic agents. To understand language in complex environments, agents must reason about the full range of language inputs and their correspondence to the world. For example, consider the scenario and instruction in Figure 1. To execute the instruction, the robot must identify the top shelf, recognize the two stacks as sets of items, compare items, and reason about the content and size of the sets. Such reasoning over language and vision is an open problem that is receiving increasing attention (Antol et al. 2015; Chen et al. 2015; Johnson et al. 2016). While existing data sets focus on visual diversity, they do not display the full range of natural language expressions, such as counting, set reasoning, and comparisons. We propose a simple natural language visual reasoning task, where the goal is to predict if a descriptive statement paired with an image is true for the image. This abstract describes our existing synthetic images corpus (Suhr et al. 2017) and current work on collecting real vision data.